AMENDMENT AND RESPONSE UNDER 37 CFR § 1.111 

Serial Number: 10/736,113 
Filing Date: December 15, 2003 

Title: SYNCHRONOUS VIDEO AND DATA ANNOTATIONS 

IN THE CLAIMS 

Please amend the claims as follows: 

1 . (Currently Amended) A surveillance system comprising: 

a camera arranged to output images of a protected area; 
an input device arranged to provide a data annotation; and 

a server arranged to synchronously store the images and the data annotation so that the 

data annotation can be used to search for a segment of the images; 

wherein the server is arranged to time stamp the data annotation; and further 

wherein the server is arranged to directly compare the time stamp of the data annotation 

to an image count when searching for the segment of the images. 

2. (Canceled). 

3. (Canceled). 

4. (Previously Presented) The surveillance system of claim 1 wherein the server is arranged 
to cause the segment of the images matching the time stamp to be displayed. 

5. (Original) The surveillance system of claim 1 wherein the server is arranged to save the 
data annotation in SQL readable form. 
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6. (Original) The surveillance system of claim 5 wherein the server is arranged to time 
stamp the data annotation. 

7. (Original) The surveillance system of claim 5 wherein the server is arranged to receive 
an SQL search string corresponding to the data annotation to be searched and to search for the 
data annotation based on the SQL search string. 

8. (Original) The surveillance system of claim 7 wherein the server is arranged to match the 
data annotation found as a result of the search to the segment of the images. 

9. (Original) The surveillance system of claim 6 wherein the server is arranged to match the 
time stamp of the data annotation to an image count when searching for the segment of the 
images. 

10. (Original) The surveillance system of claim 1 wherein the camera comprises a video 
camera, and wherein the server comprises a video server. 

1 1 . (Original) The surveillance system of claim 1 wherein the camera comprises an IR 
camera. 



12. (Original) The surveillance system of claim 1 wherein the camera comprises a thermal 
imager. 
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13. (Currently Amended) A method comprising: 
storing surveillance video in a memory; 
storing data annotations in the memory, wherein the data annotations are useful in 

searching for a video segment of the surveillance video of interest; and, 

synchronizing the stored data annotations to the corresponding video segments of the 

stored video so that the data annotations can be used to search for the video segment of interest; 
wherein the data annotations include a time stamp; and further comprising directly 

comparing the time stamp of the data annotations to an image count when searching for the 

segment of the images. 

14. (Canceled). 

15. (Previously Presented) The method of claim 13 further comprising searching for a 
particular data annotation. 

16. (Original) The method of claim 15 further comprising comparing the time stamp of the 
particular data annotation to a timing of the video when searching for the video segment of 
interest. 

1 7. (Original) The method of claim 16 further comprising displaying the video segment of 
interest that matches the time stamp of the particular data annotation. 
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18. (Original) The method of claim 13 further comprising searching the data annotations 
using a search criteria. 

1 9. (Original) The method of claim 1 8 wherein the synchronizing of the stored data 
annotations to the corresponding video segments comprises time stamping the data annotations 
with corresponding time stamps. 

20. (Original) The method of claim 19 further comprising searching for a particular data 
annotation. 

2 1 . (Original) The method of claim 20 wherein the searching includes matching the time 
stamp of the particular data annotation to a timing of the video. 

22. (Original) The method of claim 21 further comprising displaying the video segment of 
interest that matches the time stamp of the particular data annotation. 

23. (Original) The method of claim 18 wherein the searching of the data comprises using 
SQL to conduct the search. 

24. (Original) The method of claim 23 wherein the synchronizing of the stored data 
annotations to the corresponding video segments of the stored video comprises time stamping the 
data annotations with corresponding time stamps, wherein the searching comprises matching the 
time stamp associated with the stored data annotation that corresponds to the search criteria to a 
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timing of the video, and wherein the method further comprises displaying the video segment of 
interest whose timing matches the time stamp associated with the stored data annotation that 
corresponds to the search criteria. 

25. (Currently Amended) A surveillance method comprising: 
capturing images of a protected area; 

storing the images in a computer readable memory; 

storing data annotations in the computer readable memory, wherein the data annotations 
are searchable using a search criteria; and, 

storing a link that links the stored data annotations to corresponding image segments of 
the stored images so that the data annotations can be used to search for an image segment of 
interest; 

wherein the data annotations include a time stamp; and further comprising directly 
comparing the time stamp of the data annotations to an image count when searching for the 
segment of the images. 

26. (Original) The surveillance method of claim 25 further comprising searching for a 
particular data annotation using the search criteria. 

27. (Original) The surveillance method of claim 26 further comprising displaying the image 
segment of interest linked to the particular data annotation found as a result of the search. 



28. (Canceled). 
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29. (Previously Presented) The surveillance method of claim 25 further comprising 
searching for a particular data annotation using the search criteria. 

30. (Original) The surveillance method of claim 29 further comprising comparing the time 
stamp corresponding to the particular data annotation found as a result of the search to a timing 
of the images when searching for the image segment of interest. 

3 1 . (Original) The surveillance method of claim 30 further comprising displaying the image 
segment of interest whose timing matches the time stamp of the particular data annotation. 

32. (Original) The surveillance method of claim 25 wherein the search criteria comprises an 
SQL search criteria. 

33. (Original) The surveillance method of claim 32 further comprising: 
searching for a particular data annotation using the SQL search criteria; 
finding the image segment of interest linked to the particular data annotation; and, 
displaying the image segment of interest. 

34. (Original) The surveillance method of claim 25 wherein the link comprises a data 
attribute stamp, and wherein the data attribute serves as an index to retrieve video and data 
segments of the same characteristic inferred by the data attribute. 
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35. (Original) The surveillance method of claim 34 wherein the data attribute comprises 
temperature. 

36. (Original) The surveillance method of claim 34 wherein the data attribute comprises 
luminosity. 

37. (Original) The surveillance method of claim 34 wherein the data attribute comprises a 
biometric signature. 



